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Patterns and Rates of Learning in Two Problem-Based Learning Courses 


Abstract 

The concept of “practice makes perfect” was examined in this work in the context of effective learning. 
Specifically, we wanted to know how much practice was needed for students to demonstrate mastery of 
learning outcomes. Student learning patterns in two different university courses that use a similar education 
approach involving problem based learning, outcomes based assessment, and problem sequencing based on 
elaboration theory were examined. Learning outcomes for each course were explicitly defined and students 
were repeatedly assessed through sequential assignments. The cumulative proportion of criteria successfully 
demonstrated for each problem-solving attempt was determined using data retrospectively obtained from 
instructor grading records. Learning followed a typical growth pattern - it increased rapidly at first and more 
slowly with succeeding attempts. The precise shape of the learning curve differed between the two courses and 
is thought to be the result of problem difficulty and problem sequencing. Depending on these two factors, at 
least one more attempt than the number of times criteria need to be demonstrated is required and often more 
are needed to demonstrate mastery. This paper presents class-level data and future work should investigate 
individual performance and particularly why some students learn more quickly than others. Two additional 
issues for future consideration are the effect of the number of attempts on long-term retention and on the 
transferability of the learning to other problems. 

Les chercheurs se penchent sur le concept selon lequel c’est en pratiquant qu’on atteint la perfection en 
contexte d’apprentissage efficace. Plus precisement, ils cherchent a savoir a quel point les etudiants doivent 
s’exercer avant d’atteindre les objectifs d’apprentissage. Les chercheurs ont examine les modeles 
d’apprentissage des etudiants dans deux cours universitaires differents qui utilisent une approche pedagogique 
similaire axee sur l’apprentissage par problemes, revaluation des resultats et le sequengage des problemes axe 
sur la theorie de l’elaboration. Les resultats d’apprentissage ont ete explicitement definis et les etudiants ont 
ete evalues a plusieurs reprises grace a des evaluations sequentielles. Les chercheurs ont determine la 
proportion cumulative de criteres remplis pour chaque tentative de resolution de problemes a l’aide de 
donnees obtenues retrospectivement a partir des dossiers de notes de l’enseignant. L’apprentissage a suivi un 
modele de croissance typique, c’est-a-dire qu’il a tout d’abord augmente rapidement, puis ’a un rythme 
decroissant avec l’ajout de tentatives additionnelles. La forme precise de la courbe d’apprentissage etait 
differente d’un cours a l’autre et les chercheurs pensent qu’elle est attribuable a la difficulty des problemes et a 
leur sequengage. En fonction de ces deux facteurs, il faut au moins un essai de plus que le nombre requis pour 
respecter les criteres, et parfois plus, pour demontrer qu’il y a maitrise des objectifs d’apprentissage. Le present 
article presente les donnees liees a la classe dans son ensemble. Les prochains travaux devraient se pencher sur 
la performance individuelle et particulierement sur les raisons expliquant pourquoi certains etudiants 
apprennent plus vite que d’autres. Les deux autres questions a considerer sont l’effet du nombre d’essais sur la 
retention a long terme et sur la transferabilite de l’apprentissage a d’autres problemes. 
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As our personal knowledge and ability to teach have matured, we have moved to a model 
of university teaching that is grounded in three important theories that work together to allow 
and encourage effective learning. In applying these theories, a number of important questions 
about teaching and learning have surfaced and in this paper we describe and answer the 
following question related to the concept of “practice makes perfect”: How much practice 
is needed for students to demonstrate mastery of learning outcomes? 

Mastery is the philosophy that suggests that under appropriate instructional conditions, 
most students can learn “well” or master the material taught (Block & Burns, 1976; Kulik, Kulik, 
& Bangert-Drowns, 1990). In addition, it suggests that teachers can teach in such a way that 
causes mastery to occur. In this research, we define mastery as a performance level that students 
can reliably fulfill. Our definition also assumes that mastery occurs when students can repeatedly 
demonstrate the criteria that define successful performance. 

Our traditional model of teaching, one that seems common across many universities, was 
focused on presenting content, with coverage of the subject matter as the main concern. In our 
courses, we presented topics in a lecture format and a midterm and final exam were given to 
evaluate whether students could reproduce the content. We also had students work on an 
individual project toward the end of the term that required them to apply the content presented. 
Using this approach, students had three attempts to demonstrate competence and one opportunity 
for formative feedback (the mid-term exam). The feedback often reflected our overall 
impressions about the most notable characteristics of their course work. As the course 
instructors, we had not explicitly articulated learning outcomes nor were we systematically 
assessing student work using criteria referenced to these outcomes. 

The education approach we used in this work shifted from a traditional lecture-based 
model focusing on content coverage to one that relied on three theoretical constructs. First, our 
new approach used problem-based learning to engage students. Problems provided relevance and 
“drove” the learning. Second, we used elaboration theory (Reigeluth & Stein, 1983), which 
suggests that multiple attempts at problem-solving be allowed and that the respective problems 
be sequenced from less to more difficult. Third, while it has been shown that providing students 
multiple opportunities to demonstrate knowledge and develop ability can result in greater 
acquisition of learning outcomes (Ozogul, Olina, and Sullivan, 2008; Thompson & Grabau, 
2004), practice alone is insufficient. Also needed is relevant feedback from instructors. Feedback 
becomes: (a) relevant when it connects learning outcomes to student performance (Black & 
William 1998; Gibbs & Simpson 2004; Lizzo & Wilson 2008); and (b) useful when students 
have an opportunity to apply it (Gibbs & Simpson, 2004). As such, we relied on explicit and 
clear behavioral learning outcomes with aligned assessment criteria to help us provide systematic 
criterion-referenced feedback (Zundel & Needham, 2000) as our third theoretical construct. 

We applied this three-pronged educational approach to two different undergraduate 
courses at the University of New Brunswick (UNB). The first course, RCLP 1052: Mathematical 
and Economic Approaches to Problem Solving (Math/Econ), is a first year undergraduate course 
in Renaissance College, UNB’s Interdisciplinary Leadership Faculty. The course provides initial 
exposure to mathematic and economic reasoning using social or environmental problems like 
“whether a person should stay on welfare” as the basis for learning. In this course, students are 
expected to develop the skills needed to solve problems using finite mathematics and economic 
principles. The second course, FOR 3005: Silviculture and Stand Intervention Design (Forestry), 
is a third year undergraduate course in the Faculty of Forestry and Environmental Management 
and takes a design approach to silviculture. In this course, students are expected to develop 
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intervention plans for the main stages of forest stand development integrating the biology of 
growing trees, the engineering of conducting operations, and the economics of costing them. 

In addition to using a problem-based learning approach with outcomes-based assessment, 
both courses provided multiple attempts for students to master the learning outcomes. Our 
central question was “how many attempts were required”? Each attempt is a “costly” endeavor 
for both student and instructor. Since allowing too many attempts is inefficient and can waste 
student and instructor time and energy, we wanted to know the marginal return in student 
learning as a function of providing additional attempts. Students need a sufficient number to 
become proficient in any activity and instructors need a sufficient number to be confident 
students had have “mastered” the outcomes. In our judgment, students needed to accumulate at 
least two successful demonstrations of each criterion to demonstrate mastery. 

The puipose of this research was to see whether using a similar teaching approach in two 
very different courses would lead to similar learning patterns and also to identify factors that 
affect student learning. 

Background 

The two courses in this study used the three learning theories or strategies as described 
earlier - elaboration theory, problem-based learning, and outcomes-based assessment and 
evaluation. The core of elaboration theory is to select and sequence learning activities in 
increasing order of complexity in order to optimize learning. The theory uses “epitomes” that 
present the simplest form of an entire ability and then work up repeatedly by adding the factors 
or dimensions that make it more complex or challenging. This idea for sequencing and 
organizing instruction is one of the most widely used theoretical innovations in instructional 
design (ID) (Wilson & Cole, 1996). In total, the theory consists of seven major strategic 
components (Reigeluth & Stein, 1983): (a) an elaborative sequence, (b) learning prerequisite 
sequences, (c) summarizers, (d) synthesizers, (e) analogies, (f) cognitive strategy activators, and 
(g) a learner control format. These seven strategies are briefly explained in Table 1. 
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Table 1 


Reigeluth and Stein’s Elaboration Theory ( 1983) 


Strategic Component 

Description 

Elaborative Sequence 

Simple to complex sequence in which the first lesson 
epitomizes the ideas and skills to follow. This is 
considered the most critical component of elaboration 
theory. 

Learning Prerequisite 
Sequences 

Elaborates upon organizing the content from the first 
strategy. The epitomes are sequenced according to the 
order of the steps: forward chaining, backward chaining, 
hierarchical sequencing, general to detailed sequencing 
and simple to complex sequencing. 

Summarizers 

In order to review what has already been learned, a 
“summarizer” is created. A summarizer provides a 
concise statement of each idea. Two types are used: 
internal and within-set. 

Synthesizers 

Integrates and interrelates the ideas taught thus far in 
order to facilitate deeper understanding, meaningfulness, 
and retention in regards to the content area. 

Analogies 

The use of a familiar idea or concept to introduce or 
define a new idea or concept. 

Cognitive Strategy Activators 

Imbedded activators use pictures, diagrams and other 
element to force the learner to interact with the sequence 
and content. Detached activators cause the learner to 
employ a previously acquired cognitive skill. 

Learner Control Format 

Allows the learner to control the selection and sequencing 
of instructional elements (e.g. content, rate, components 
and cognitive strategies). 


This paper focuses on the elaborative sequence component. The elaborative sequence 
concept suggests that instruction follows a simple to complex sequence with early lessons 
containing fundamental ideas at a concrete skill level. Layers of complexity are then added 
sequentially. The epitomes used in both courses were the problems we asked students to solve. 
Consistent with elaboration theory’s concept of epitomes, the whole outcome needed to be 
expressed in each problem rather than simply some parts. As such, each problem was a complete 
task that called on the use of all components of an outcome to be learned. The courses differed in 
one important respect; in the Math/Econ course, the first three problems increased gradually in 
terms of complexity and the fourth problem was a rewrite of one of the first three (usually the 
one on which students did most poorly in a first attempt). In the Forestry course, the second 
problem was significantly more difficult than the first, third or fourth problems. In other words, 
while the Math/Econ course had a smooth developmental sequence that flattened near the end of 
the course, the Forestry course had a developmental “hump” with the 2 nd problem. That problem 
required decisions about which tress to leave and which ones to harvest, which has complicated 
biological, operational and economic interactions. 

Both courses used a problem-based learning (PBL) approach. PBL focuses on problem 
solving with the problem “driving” the learning (Matthew and Hughes, 1994). The problem 
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statement clearly describes the issue, however it is the student that must clearly define and solve 
the problem. The learning that occurs in PBL is developmental in that there is sequencing and 
building upon already established skills as the difficulty of the problem increases. This approach 
is an effective way for students to learn to solve problems by experiencing the process as active 
participants. PBL is more than problem solving and, as suggested by Matthew and Hughes 
(1994), a key element of PBL is the learning that students do in addressing the problem. In this 
regard, while our primary goal was for our students to leam a problem-solving process, we also 
wanted them to learn disciplinary “content”. Dym, Agogino, Eris, Frey, and Leifer, (2005) and 
Matthew and Hughes (1994) suggest problem-based learning as a method for developing higher- 
level cognitive, personal and interpersonal skills. Another strength of PBL is the ability of 
students to extend what has been learned in one context to other, new contexts (Dym et al., 
2005). While the outcomes related to personal and interpersonal skills were not specific 
outcomes for these courses, they were general objectives of the degree programs as a whole. 

Outcomes-based assessment means setting explicit learning outcomes and assessing 
student performance directly against them. Outcomes describe the knowledge, skills and 
attitudes we want students to acquire and demonstrate by the end of the course (Zundel & 
Needham, 2000). It is essential that outcome statements are explicit, hierarchically organized, 
behaviorally described and given to students (Zundel & Needham, 2000). In both courses, the 
syllabi described the outcomes in behavioural terms. For example, in the Math/Econ course, one 
of the outcomes was defined as: “Apply a structured problem-solving framework effectively to 
solve financial, economic and sustainability problems.” Outcomes were then decomposed into 
behavioral criteria. Tables 1 and 2 give the outcomes and criteria used in the two courses. The 
learning outcomes and criteria for each course were explicit, behavioral and hierarchical. When 
students repeatedly fulfill the criteria, we assume they have mastered the outcome. 

In the Math/Econ course, students were provided four attempts to demonstrate 
competence in the learning outcomes. These attempts took the form of increasingly difficult 
problems (based on elaboration theory). The first problem provided the students with financial 
data for a single parent family on income assistance. Their goal was to determine whether it was 
financially more attractive for the breadwinner of this family to go to work or remain on welfare. 
The focus of the first problem was building a financial model and learning the dimensions of the 
structured problem-solving framework. The technical issues were very simple and the decision 
variables (ending bank balance) were given to the students. In the second problem, students were 
again provided the decision variable (present net worth and benefit/cost ratio of a logging 
business investment), but were required to do some research about the nature of the problem 
(e.g., compare horse logging to mechanized logging, examine conceptual ideas like the time- 
value of money that are used to calculate present net worth). Again, students built a model and 
applied the problem-solving framework. In the third problem, students were asked to develop 
their own indicators of social, economic and environmental sustainability to move from an 
abstract concept like sustainability to concrete mathematical indicators for a problem of their 
choice. They again had to build a model and apply the problem-solving framework. Finally, the 
fourth problem was a rewrite of one of the prior three. After reporting their solution to each 
problem, students were provided summative and formative feedback on their report which was 
referenced to the criteria established at the beginning of the course for each learning outcome. 

The Math/Econ course had three learning outcomes: (a) problem-solving (PS), (b) 
mathematical modeling, and (c) economics knowledge. The PS outcome had nine criteria, the 
mathematical modeling outcome had five, and the economics knowledge outcome had eight 
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criteria. These criteria ranged from simple ones like “identify objectives, constraints and 
resources [of the problem]” to higher order ones like “identify assumptions and predict their 
effects on the solution”. Table 2 gives a complete list of the criteria for the PS outcome for this 
course. The students had to demonstrate each criterion in at least two separate attempts to 
indicate their mastery of each. The course grade was based on the percent of mastered criteria, 
where each was weighted according to its relative importance. 


Table 2 

Math/Econ Problem-Solving (PS) Outcome and Associated Criteria 


PS Outcome 


Criterion Description 


By the end of this course, 
students will be able to apply 
a structured problem-solving 
framework effectively to 
solve financial, economic and 
sustainability problems. 


1. Rephrase or describe problem accurately. 

2. Identify constraints, objectives and resources. 

3. Define key terms subject to ambiguity. 

4. Define several possible solutions. 

5. Predict performance of each solution in terms of 
objectives and constraints. 

6. Select optimal solution. 

7. Identify assumptions and predict their effects on the 
solution. 

8. Reflect on what parts of solution process or product 
were best/worst. 

9. Identify what personal skills or /knowledge limited or 
/helped most during implementation. 


The Forestry course had four outcomes: (a) effective writing (EW), (b) primary 
information acquisition (PIA), (c) stand intervention planning (SIP), and (d) display of stand 
intervention technical knowledge (K). Stand intervention planning has 18, effective writing has 
eight criteria, primary information acquisition has five, and display of stand intervention 
technical knowledge has seven. As an example, Table 3 gives a complete list for the SIP 
outcome for the Forestry course. 


Produced by The Berkeley Electronic Press, 2012 


5 



The Canadian Journal for the Scholarship of Teaching and Learning, Vol. 3 [2012], Iss. 1, Art. 4 


Table 3 

Forestry Stand Intervention Planning (SIP) Outcomes and Associated Criteria 


SIP Outcome 


Criterion Description 


By the end of this course, the 
student will be able to 
construct a site specific SIP 
that incorporates all relevant 
criteria and is specific to the 
issue presented. 


1. Frame problem by providing goals. 

2. Transform goals into objectives. 

3. Describe stand and site condition. 

4. Describe intervention. 

5. Describe implementation. 

6. Identify and describe ancillary resources. 

7. Predict resource consumption. 

8. Predict immediate benefits. 

9. Illustrate yield prediction. 

10. Describe primary and secondary impacts. 

11. Describe monitoring plan. 

12. Stand description is reasonable. 

13.Intervention is reasonable. 

14. Implementation methods are reasonable. 

15. Resource consumption is reasonable. 

16. Immediate benefit prediction is reasonable. 

17. Yield prediction is reasonable. 

18. Monitoring plan is reasonable. 


Students learned and demonstrated competence in the Stand Intervention Planning 
criteria through four problems. The first problem was the simplest and required students to 
develop a stand intervention plan (something akin to a doctor’s prescription for a patient but 
aimed at treating a forest area) for a pre-commercially thinned, 15-year old, dense stand of trees. 
The second, a much more complicated problem required a commercial thinning plan, the third 
was a regeneration plan, and the fourth problem was a harvest plan. The third and fourth 
problems have a similar, but slightly lower, degree of difficulty than the second problem which 
is technically the most difficult. This problem difficulty sequence does not strictly follow 
elaboration sequencing theory but was created to ensure students encountered the concept of 
commercial thinning early in the term for logistical reasons and before they had accumulated 
enough criteria which would exempt them from it. Given that students only need to complete 
problems until they have mastered criteria, it is possible that some students will not attempt 
problems given late in the course as they will have demonstrated the criteria the required number 
of times in earlier problems. This represents a trade-off of the principles of gradual incline in 
difficulty suggested by elaboration theory in favour of a curricular need to ensure mastery of key 
content. 

Assessment of student learning in both courses and corresponding instructor feedback 
consisted of examining student reports in terms of the criteria that define each outcome. It has 
been shown that assessment and appropriate feedback need to show the student how s(he) is 
progressing in ways that “connect the judgment of student performance with the outcomes 
students are expected to acquire (Alverno College, 1994).” When each assignment was given, 
the assessment criteria were provided and subsequent feedback was referenced directly to them. 
Students earned grades by demonstrating mastery of the criteria (i.e., that they had fulfilled the 
required criteria at least two times). Once a student had demonstrated the criteria twice they did 
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not have to demonstrate them again, although solving the problems to demonstrate the remaining 
criteria often required others previously demonstrated be addressed again since many criteria are 
inter-related. When students were not able to demonstrate a criterion twice, they did not receive 
the marks associated with it. 

Methods 

In this paper, we show the data for the most important outcome in each course (the 
outcome with the highest grade weighting). These are also the ones with the greatest 
“granularity” of information about student performance. That is, problem solving for Math/Econ 
and stand intervention planning for the Forestry course had the largest number of criteria and 
provided the greatest resolution of information relative to the questions we wanted to answer. 
Including the two courses in this research allowed us to see whether using a similar teaching 
approach in two very different courses would lead to similar learning patterns and it allowed us 
to identify factors that affect student learning that we might not see otherwise. 

Class records were obtained from instructor grading records for two cohorts for the 
Math/Econ course (academic years 2003 and 2004), and four cohorts for the Forestry course 
(2000 to 2003 academic years). The records were stripped of their student names and the 
following variables were calculated for each attempt: (a) cumulative average proportion (%) of 
criteria successfully demonstrated at least once, and (b) cumulative average proportion (%) of 
criteria demonstrated at least twice (for attempts 2-4). The calculations involved pooling data 
for all the cohorts for each course (i.e., two for the Math/Econ course and four for the Forestry 
course). The courses were not modified to carry out this research; rather aggregate class data 
from offerings 5-10 years prior to the research were used. No linkage was possible between the 
records used and individual students. The consequence of this approach is that the data illustrates 
class mastery rather than individual student mastery of the criteria. This level of analysis 
informs decisions about how to conduct a class rather than providing information about 
individual student performance. 

The data were examined using Minitab Statistical Software (Minitab Inc., 2010) 
graphically and using an analysis of variance (ANOVA) with Tukey post-hoc analysis to 
determine if differences in the proportion of criteria students demonstrated between attempts was 
statistically significant. The alpha level was set to 0.05. 

Results 

The results for both courses are illustrated in Figures 1 and 2 with the corresponding data 
in Table 4. Figure 1 illustrates the percentage of criteria successfully demonstrated for each 
problem for each course when students were required to demonstrate the criteria one time (IX), 
and Figure 2 provides similar information when criteria needed to be demonstrated twice (2X). 
We refer to the problems as “Attempts” in this analysis. 
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Figure 1. One time demonstration pattern for the PS and the SIP Criteria. PS - Problem Solving outcome 
for the Math/Econ course, SIP - Stand Intervention Planning outcome for Forestry course. 



Figure 2. Two-time demonstration pattern for the PS and SIP criteria. PS - Problem Solving outcome for 
the Math/Econ course, SIP - Stand Intervention Planning outcome for Forestry course. 
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Table 4 


Percentage of Abilities Demonstrated Once (IX) and Twice (2X)for Outcomes of Both Courses 


Attempt Number 


Mean + Standard Deviation 



Problem-Solving Ability 
(Math/Econ) 

Stand Intervention Planning 
Ability (Forestry) 


IX 

2X 

IX 

2X 

1 

61.1+25.8 

— 

40.4+17.9 

— 

2 

91.0+16.3* 

49.6 + 24.8 

67.6 + 18.9* 

25.9+15.5* 

3 

94.4 + 14.1* 

80.3 + 22.3* 

82.8 + 17.4* + 

49.8 + 22.7* + 

4 

96.6 + 11.7* 

85.5 + 21.7* 

88.0+ 18.2* + 

68.4 + 27.8 * + 


* - Indicates statistically significant difference with first attempt (p<0.05) 

+ - Indicates statistically significant difference between Attempts 2-4 (p<0.05) 


The ANOVA was conducted to determine if there were differences in the proportion of 
criteria demonstrated between attempts. Results of the ANOVA are found in Table 4. 

Post hoc analysis revealed that for the Math/Econ data, there were differences between 
Attempt 1 and all other attempts (p=0.000), however there were no significant differences 
between Attempts 2, 3 and/or 4. There were statistically significant differences between Attempt 
2 and both attempts 3 and 4 (p=0.000) for the 2X demonstration. However, attempts 3 and 4 
were not statistically significant from one another. 

For the Forestry data, the ANOVA results showed that there were differences between all 
four attempts. Attempt 1 was statistically different from all other attempts (p=0.000). In addition, 
the post hoc analysis revealed that there were statistically significant differences between 
Attempts 2 and each of 3 (p=0.0004) and 4 (p=0.000) for the IX demonstration. There were 
statistically significant differences between Attempts 2, 3 and 4 (p=0.000) for the 2X 
demonstration. 

Figure 1 shows that 61.1% of the criteria in the Math/Econ course were successfully 
demonstrated on Attempt 1 and this increased to 91.0 % at Attempt 2. There were slight 
increases in the third and fourth Attempts (problems 3 and 4) to 94.4% and 96.6% respectively, 
but these were not statistically significant. In other words, by the fourth problem, on average, 
students had demonstrated 96.6% of the criteria at least once. 

The Forestry course had a similar pattern although the demonstration curve shifted down. 
The percentage of criteria demonstrated in the first Attempt (Problem 1) was 40.4%, which then 
increased to 67.6% on Attempt 2, 82.8% on Attempt 3, and to 88% on the 4 th Attempt. The 
difference between Attempt 3 and Attempt 4 was not significant when criteria were 
demonstrated one time. 

Figure 2 shows the proportion of criteria demonstrated at least twice - in other words 
“mastered”. Obviously, this variable only has meaning after the second attempt. By increasing 
the required number of successful demonstrations from one to two, the degree of certainty that a 
student has in fact ‘mastered’ the criteria increases. In the Math/Econ course, students were able 
to demonstrate 49.6% of the criteria a second time on Attempt 2. Interestingly, 11.5 % fewer 
criteria were demonstrated on the second attempt than originally demonstrated during the first 
attempt (61.1 vs. 49.6, see Table 4). In effect, while students demonstrated more criteria on the 
second attempt overall, they failed to demonstrate 11.5 % of them a second time. By the Attempt 
3, the number of criteria demonstrated twice increased 30% to 80.3%, and by the 4 th attempt 85.5 
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% of the criteria had been demonstrated twice. This is 11.1 % less (85.5% vs. 96.6%) than the 
number demonstrated one time. It should be noted that the percent of criteria demonstrated 
between the third and fourth attempts is not statistically significant. 

In the Forestry course, students demonstrated 25.9 % of the criteria twice by Attempt 2, 
which was 14.5 less than on Attempt 1 (40.4% vs. 25.9%, see Table 4). By Attempt 3, the 
percentage increased to 49.8%, and by Attempt 4, 68.4% of the criteria had been demonstrated 
twice by the class, down 19.6 % from the number demonstrated once. Differences in the average 
student performance were statistically significant for all attempts. 

Discussion 

Our results show similarities in the learning patterns in the Math/Econ and Forestry 
courses despite different course content. That is, students’ ability to demonstrate the problem 
solving outcome in the Math/Econ course and the stand intervention planning outcome in the 
Forestry course grows rapidly in the first two attempts, and then tapers off as they get more 
practice with course content and the problem-solving tasks. Despite the courses’ general 
similarity in pedagogical approach, there are specific differences in the learning curves between 
the two courses (Figures 1 and 2). One difference is the location of the curve. The Forestry 
performance curve is lower than the Math/Econ course curve (Figures 1 and 2). In effect, the 
amount of learning in the Forestry course relative to the expectation as defined by the criteria is 
lower than in the Math/Econ course. The second difference is the precise shape of the two 
curves. The rate of learning differs between the two courses with students demonstrating criteria 
faster in the Math/Econ course than in the Forestry course. 

We believe that there are several factors that affect the location and shape of the learning 
curves. The primary ones are the inherent difficulty of the problems and the timing or sequencing 
in which increasingly difficult problems are presented to students. Relatively speaking, the 
Forestry problems appear to be more difficult for the forestry students than the Math/Econ 
problems are for the Renaissance College students. This may be due to the fact that the forestry 
problems require specialized technical knowledge and are more complicated because they 
involve more and a wider spectrum of types of information (e.g. biological, financial and 
operational data) than the Math/Econ problems. 

The sequencing of problems also affects student success. Increasing problem difficulty at 
too fast a rate stifles or reduces the number of criteria students demonstrate. Problem 2 is an 
example of this in the Forestry course and it becomes evident when students needed to 
demonstrate a criterion a second time. In this course, the second problem is much more difficult 
than the first and more difficult than the third. This is in contrast to the Math/Econ course in 
which problem difficulty rises smoothly through the first three problems before stabilizing at the 
fourth. What the results show is that the type of sequencing used in the Math/Econ course is 
associated with a pattern in which mastery occurs more quickly with a plateau reached earlier 
compared to the Forestry course. The latter had the most difficult problem placed second and 
showed a “shallower” learning curve through the second problem. 

The challenge that students face with a difficult problem is amplified when two 
demonstrations are required (see Figure 2). If students are marginally competent at a given 
criterion, it is more likely that they will fall below the minimal competence if they have to 
demonstrate it several times using different theoretical and technical content. For example, a 
Math/Econ student who successfully demonstrates her ability to predict the financial 
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attractiveness of multiple solutions to a given problem may fail, in the next problem, to do the 
cognate task of assessing the sustainability of several problem solutions because of the latter’s 
greater technical complexity. Table 4 shows that student learning in the Math/Econ course 
increased from 49.6% in Attempt 2 to 80.3% at the conclusion of Attempt 3 and this increase of 
30.7% is 6. 0% greater than the 23.9% increase observed with student performance between the 
second and third Forestry problems (25.9% to 49.8% respectively). 

While elaboration theory suggests increasingly difficult problems need to be used, 
Scardimalia and Bereiter (1994) indicated that as expertise develops, skills and concepts that 
were initially challenging become routine, thereby freeing up mental capacity for further 
development. One of our challenges as teachers is to determine if and how we can enhance the 
rate of learning as students progress through the course, and maintaining an appropriate 
developmental incline is one component of this. 

In both the Forestry and Math/Econ courses, prompt criterion-referenced feedback was 
provided to students. We believe that the substantial increase in the number of abilities 
demonstrated between the first and second attempts is partly due to the feedback students receive 
relative to the criteria. They learn what they need to “know and do” to be successful and try to do 
it since there is close alignment between the criteria, the outcomes, and the grading scheme. This 
would be consistent with the findings of Black and William (1998) and Gibbs and Simpson 
(2004). 

Repeated demonstration of the criteria was important for us to be comfortable that 
students had reliably mastered the outcomes. The ‘percentage of criteria demonstrated' data 
presented for both courses showed differences in criterion mastery when the standard for mastery 
required two demonstrations rather than one. Figure 2 shows that for the Forestry course, the 2X 
learning curve is linear and continues to increase without a plateau or leveling off. This suggests 
that more attempts may be needed so students can integrate what they know and focus on 
fulfilling criteria they struggle with. 

We can conclude from the results of our analysis that providing only two attempts to 
master learning in both the mathematics and forestry courses is insufficient and would result in 
students demonstrating a much lower level of ability by the end of the course than providing four 
attempts. With only two attempts (e.g. a mid-term and final examination), students in Math/Econ 
would have demonstrated 91.0% (average) of criteria at least once but only 49.6% twice. It took 
Math/Econ students on average four attempts to reach 85.5% of criteria demonstrated twice, 
though the difference between Attempt 3 and Attempt 4 was not statistically significant. This 
implies substantial growth and more certain mastery of the criteria in the third attempt and less 
so in the fourth. A different pattern is evident for Forestry, where the average student 
demonstrates 88.5% of the criteria once for the SIP outcome and 68.4% twice after four attempts, 
with statistically significant growth on all attempts when two successful demonstrations are 
required to achieve mastery. 

Our results indicate that in some cases, mastery of a high proportion (e.g., over two- 
thirds) of the criteria requires at least four and maybe five attempts when mastery is defined as 
demonstrating criteria twice. Clearly, the rate of getting to mastery is affected by problem 
difficulty and sequencing. Although not obvious by the data presented here, student performance 
was also affected by the difficulty of the individual criteria. Anecdotal student comments 
indicated that certain criteria in both courses were simply harder to understand and fulfill than 
others, and more attempts to reliably demonstrate them were required. For example, in the 
forestry course, providing “reasonable yield predictions” was difficult for many students. For 
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those criteria where the standards are too high, student performance starts off lower and grows at 
a slower rate. As such, one of our challenges as instructors of this course is to establish criteria 
that are challenging but attainable given the time, energy, and constraints faced by both students 
and instructors. Some of the ways we can do this are by simplifying, clarifying or illustrating the 
criteria more fully, reducing the expected performance level by relaxing our standards, being 
more precise about criteria so students know exactly what to do, or subdividing criteria into more 
sub-criteria. This latter approach was used in the Forestry course and had been integrated into the 
course syllabus before the period studied here. The number of criteria was doubled with half 
being inclusion of an item and half being a requirement that those inclusions be “reasonable” 
(Table 2). While this increased the number of criteria students needed to demonstrate, it also 
reduced complexity of individual criteria and was simpler for students. A final strategy can be to 
reduce the number of learning outcomes dealt with in the course to allow for more time and 
effort to be devoted to the remaining ones. 


Conclusion 

Overall, we found both similarities and differences in learning patterns from two 
dissimilar courses in two unrelated faculties with very different learning outcomes and students. 
One of the similarities was that students’ ability to demonstrate outcome criteria increased 
rapidly at first and then declined with an increasing numbers of attempts. Differences included 
the presence of a plateau in student learning after Attempt 3 in the Math/Econ course and the 
absence of a plateau in the Forestry course. Additionally, the absolute level of the student 
performance was higher for the Math/Econ course and, finally, that the slope of the Math/Econ 
course was initially higher than for the Forestry course. 

Course-related factors which may have affected the precise shape of the performance 
curve included problem difficulty, problem sequence, criteria specification, and the number of 
times students had available to demonstrate them. Since the students were different in the 
Math/Econ and Forestry courses, performance differences may simply be due to inherent 
differences in the students themselves. 

Difficult problems limit the number of criteria students demonstrate and, as a 
consequence, more attempts are required for students to reach mastery. Problem sequencing 
interacts with problem difficulty to affect mastery. In the Math/Econ course, where problem 
sequencing followed a smooth developmental incline through the first three problems followed 
by a leveling off in the fourth problem was associated with a more rapid and higher level of 
mastery before a plateau was reached than in the Forestry course which had a more difficult 
problem placed earlier in the course (Problem 2) and was associated with lower and slower 
learning. However, the difference in demonstrated ability in the two courses is evident from the 
very first problem and therefore cannot entirely be attributed to the second problem in the 
Forestry course. It is possible that the first problem in the Forestry course was inherently more 
difficult than the first problem in the Math/Econ course. It is also possible that the two student 
populations also had inherent differences such as entering grade point averages. Fastly, the 
forestry course required a higher level of prior technical knowledge and was placed in the third 
year whereas the Math/Econ course was placed in the first year and had no specific prerequisites. 
Further research would be required to tease out the relative importance of these features and the 
interactions among them. However, the results presented here suggest that at least five attempts 
will be needed to reach a plateau in the Forestry course. Providing fewer attempts clearly means 
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some criteria will not be mastered by a significant proportion of the class (e.g., one-third). Three 
attempts were needed in the Math/Econ course. 

The finding that sufficient attempts are needed in order for students to demonstrate 
mastery learning is consistent with the literature on the role of practice in student learning in 
higher education (e.g., Myers & Myers, 2007). It would be helpful in subsequent research to 
determine why and how additional practice helps students learn, something that this study by its 
quantitative nature could not assess. This would involve an investigation of how students 
experience our approach to learning, and which of the course structures and practices lead to the 
learning that is reflected in increasing mastery. 

In this paper, we have presented class-level data only. It would also be interesting to 
investigate the distribution of individual performance in the class. For example, it would be 
interesting to know why some students learn very quickly, some more slowly, and some almost 
not at all. This could point the way to course designs in which individual students receive 
additional help when they demonstrate slow initial growth patterns. 

Two other important questions that this work raises are the issue of long-term retention 
and transferability. Future work should critically and systematically examine whether and to 
what extent this is achieved. Specifically, the question is whether the probability of 
demonstrating the criterion in repeated attempts in one course is a good predictor for the ability 
of the student to demonstrate the criterion again later (e.g., in five years) in similar contexts or to 
transfer it to other domains (e.g., in community problem solving rather than economic decision 
making). 
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